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[57] 



ABSTRACT 



A method and device for preprocessing streams of encoded 
data (e.g. compressed in accordance with in MPEG 
standard) to permit a decoder to decode the streams, back- 
to-back (Le., one stream immediately following another), 
without being reset and without producing video artifacts. 
The present invention includes verifying that a multiplexed 
stream complies with an encoding standard and preprocess- 
ing packets of the packeliad encoded video sequence such 
that no video artifacts arc produced when a video decoder 
decodes an adjacent encoded video sequence. 

17 Claims, 16 Drawing Sheets 
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video sequenceO { 300 


No. of bits 


Mnemonic 


next_start_codeO 






sequence headerO 302 






if (nextbitsO = extension_start_code) { 






sequence extensionO 304 






do{ 






extension and user datafO^ 306 






do { 






if (nextbitsO = group_start_code) { 






308 


group_ofjrictures_headerO 






310 


extension_and_user_data( 1 ) 






} 






312 


pictureJieadeiO 






314 


picture^coding^extensionO 






316 


extensions_and_user_data(2) 






318 


picture_dataO 






} while ( (nextbitsO=picture_start_code) II 






(nextbitsO = group_start_code) ) 






if (nextbitsO! = sequence_end_code) ( 






sequenceJheaderO 






sequence_extensionO 






} 






} while(nextbitsO!=sequence_end_code) 






} else { 






/* ISO/EEC 11172-2 */ 






1 






sequence_end_code 320 


32 


' bslbf 


} 
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pictureJieadeiO { 


312 


No. of bits 


Mnemonic 


picture_start_code 


502 


32 


bslbf 


temporaljreference 


504 


10 


uimsbf 


picture coding: type 


3 


uimsbf 


vbv_delay 


16 


uimsbf 


iffpicture codine type = 2 II picture codinetype = 3) { 






full _pel forward vector 


1 




forward _f_code 


3 


uimsbf 


} 






if (picture_coding_type = 3) ( 






full_pelJ)ackward__vector 


1 




backward_f_code 


3 


uimsbf 


} 






while (nextbitsO = T) { 






extra_bit_picture /* with the value T */ 


1 


uimsbf 


extra_information_picture 


8 




} 






extra_bit_picture /* with the value '0' */ 


1 


uimsbf 


next_start_codeO 






} 







FIG. 5 (PRIOR ART) 



group_ofj)ictures_headeiO { 


308 


No. of bits 


Mnemonic 


group_start_code - — ^— - 
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32 


bslbf 


time_code 
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bslbf 


closed_gop — \ — - 


•602 
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uimsbf 


brokenjink 
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next_start_codeO 
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METHOD AND DEYICE FOR 
PREPROCESSING STREAMS OF ENCODED 
DATA TO FACILITATE DECODING 
STREAMS BACK-TO BACK 

BACKGROUND OF THE INVENTION 
a. Field of the Invention 



b. Related Ait 

The MPEG standard focuses on the encoding and trans- 
port of video and audio data. In general, the MPEG standard 
uses compression algorithms such that video and audio data 
may be more efficiently stored and communicated. 

The International Organization for Standardization (or the 
Organization International Dc Normalisation) (hereinafter 
referred to as "the ISO/IEC") has produced the MPEG II 
standard for the coding of moving pictures and associated ^ 
audio. This standard is set forth in four documents. The 
document ISO/TEC 13818-1 (systems) specifies the system 
coding of the specification. It defines a multiplexed structure 
for combining audio and video data and means of repre- 
senting the timing information needed to replay synchro- 30 
nized sequences of the audio and video data in real-time. The 
document ISO/IEC 13818-2 (video) specifies the coded 
representation of video data and the decoding process 
required to reconstruct pictures. The document ISO/IEC 
13818-3 (audio) specifies the coded representation of audio 35 
data and the decoding process required to reconstruct the 
audio data. Lastly, the document ISO/IEC 13818-4 
(conformance) specifies procedures for determining I he 
characteristics of coded bitstreams and for testing compli- 
ance with the requirements set forth in the ISO/IEC docu- ^ 
ments 13818-1, 13818-2, and 13818-3. These four 
documents, hereinafter referred to, collectively, as "the 
MPEG U standard" or simply "the MPEG standard", are 
incorporated herein by reference, 

A bit stream, multiplexed in accordance with the MPEG 45 
standard, is either a "transport stream" or a "program 
stream'*. Both program and transport streams are constructed 
from "packetized elementary stream" (or PES) packets and 
packets containing other necessary information. A "pack- 
etized elementary stream" (or PES) packet is a data structure 5Q 
used to carry "elementary stream data". An "elementary 
stream" is a generic term for ooe of (a) coded video, (b) 
coded audio, or (c) other coded bit streams carried in a 
sequence of PES packets with one and only stream ID. 
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the video bitstream measured in uoits of 400 bits/second. A 
twelve (12) bit rate extension field may be included in the 
extension and user field 310. RepeaLing the sequence header 
in the video bitstream allows data elements of the initial 
5 sequence header to be repeated to permit random access into 
the video sequence. 

As shown in FIG. 5, the picture header 312 includes a 32 
bit picture start code field 502, as well as a ten (10) bit 
temporal reference field 504. The temporal reference field 
10 504 is a unsigned integer associated with each input picture. 
This integer is incremented by one, modulo 1024, for each 
input frame. If a frame is coded as two interleaved fields, the 
temporal reference 504 in the picture header 312 of both 
RG. 2 is a diagram which illustrates the syntax of i PES ^lds is the same. Following a group start code 604 in the 
packet 200 15 S^P °^ pi ctures header 308, the temporal reference field 

504 is reset to zero. 

FIG. 7 is a high level block schematic showing a system 
for encoding, communicating, and decoding video and audio 
data in accordance with the MPEG II standard. 

20 



25 



30 



35 



The pay load 210 of a PES packet 200 may carry a 
sequence of video frames or audio frames, for example. 
FIGS. 3<2 and 36 illustrate the high level organization of a 
video bitstream (or video sequence) 300 in accordance with 45 
the MPEG II standard. As shown in FIGS. 3a and 3&, the 
video bitstream (or video sequence) 300 includes a sequence 
header 302, which maybe followed by a sequence extension 
field 304. The sequence extension field 304 may be followed 
by an extension and user field 306, which may be followed 50 
by a group of picture header 308, and optionally, another 
extension and user field 310. Iq any event, a picture header 
312 follows the sequence extension field 304 (in addition to 
any of the other fields). Apicture coding extension field 314 
follows the picture header field 312. An optional extension 55 
and user field 316 follows. Next, the picture data 318 is 
provided. More sequences may be provided. Otherwise, the 
video sequence 300 is ended with a sequence end code 320. 
Each section of the video sequence 300 is described in the 
MPEG U standard. However, for the reader's convenience, $0 
the sections particularly relevant to the present invention are 
described below 

As shown in FIG. 4, the sequence header 302 includes a 
32 bit sequence header code field 402 and an eighteen (18) 
bit rate value field 404. The sequence header code field 402 65 
is 000001 B3 hex and identifies the beginning of a sequence 
. header. The bit rate value field 404 identifies the bit rate of 



The transport stream multiplexer 712 multiplexes the 
encoded audio and video packets to form a transport stream 
100 and provides the transport stream 100 to communica- 
tions link 722. At a remote end of the communications link 
722, a transport stream demultiplexer 724 receives the 
multiplexed transport stream 100. 

Based on the packet identification (or PID) number 114 of 
a particular packet, the transport stream demultiplexer 724 
separates the encoded audio and video packets and provides 
the video packets to a video decoder 730 via link 728 and the 
audio packets to an audio decoder 734 via link 732. The 
transport stream demultiplexer 724 also provides timing 
information to a clock control unit 726. The clock control 
unit 726 provides timing inputs to the both the video decoder 
730 and the audio decoder 734 based on the timing infor- 
mation provided by the transport stream demultiplexer 724. 
The video decoder 730 provides decoded video data which 
corresponds to the video data originally provided to the 
video encoder 702. Similarly, the audio decoder 734 pro- 
vides decoded audio data which corresponds to the audio 
data originally provided to the audio encoder 714. 

As mentioned above, transport streams 100 permit one or 
more programs with one or more independent time bases to 
be combined into a single stream. That is, a transport stream 
100 may include a first program and a second program. In 
presently contemplated systems, both the video decoder 730 
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and the audio decoder 734 must be reset before decoding a 
aext program, for reasons which will be explained below. 
Thus, for example, in such systems there must be a temporal 
gap (e.g., one second) between the decoding of the first and 
second programs to permit the video and audio decoders 730 
and 734, respectively, to be reset. This temporal gap pre- 
cludes the playing of the secood program directly following 
the first program. Moreover, it is difficult to determine when 
one program ends and another begins in real-time. Thus, a 
method and/or a device is needed to permit more than one 
program to be played (i.e., decoded) back-to-back. The 
method and/or device should also overcome, or avoid, the 
difficulties of detenmning program boundaries in real-time. 



,678 

6 

an audio sequence and a video decoder can, without being 
reset, decode the encoded video sequence Uj produce a video 
sequence. The method of the present invention (i) verifies 
that the multiplexed stream complies with an encoding 

5 standard, (ii) preprocesses packets of the packetized, 
encoded, video sequence such that do video artifacts arc 
produced when the video decoder decodes an adjacent 
encoded video sequence, and (iii) preprocesses the packets 
of the packetized, encoded, audio data sequence such that its 

1Q start time is within a first predetermined time of the start 
time of the video sequence and its temporal length is within 
a second predetennined time of the temporal length of the 
video sequence. 
The step of preprocessing the packets of the packetized, 

^ encoded, video sequence preferably includes (i) deleting any 
video frames that cannot be decoded if video frames of the 
video sequence are not temporally correct, and (ii) deleting 
any video frames following a code indicating an end of the 
encoded video sequence. The step of preprocessing the 

20 packets of the packetized, encoded, audio sequence prefer- 
ably includes (i) removing any partial audio frames, (ii) 
adjusting (Le., adding or deleting) the number of audio 
frames, if necessary, such that the audio and video sequences 
start within the first predetennined time, and (iii) adjusting 

22 (i.e., adding or deleting) the number of audio frames, if 
necessary, such that the temporal lengths of the audio and 
video sequences are within the second predetermined time. 



30 



35 



40 



Thus, as can be appreciated from the above discussion of 
MPEG and MPEG H video encoding, the video decoder 730 
often needs past and' future frames to decode a picture 
(B-Picturc) frame. If die last temporal (displayed) frame of 
a first program is used in de coding a first temporal 
(displayed) frame of a second program, or if a partial video 
frame is used, the output of the video decoder 730 will have 
been improperly decoded, disadvantageous! y causing video 
artifacts. Thus, as discussed above, the decoders must be 
reset between programs in known systems. 



SUMMARY OF THE INVENTION 



Specifically, the present invention provides a method for 
preprocessing multiplexed streams of packets of packetized, 65 
encoded, audio and video sequences such that an audio 
decoder can decode the encoded audio sequence to produce 
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FIG. 6 illustrates the structure of a group of pictures 
header of the MPEG II video sequence of FIGS. 3a and 36. 

FIG. 7 illustrates an encoding, transmission, and decoding 
system envisioned by MPEG II. 
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BRIEF DESCRIPTION OF TIIE DRAWINGS 



FIG. 2 is a diagram which illustrates the syntax of an ^ 
MPEG II PES packeL 

FIGS. 3d and 3b illustrate the organization of an MPEG 
II video sequence. 

FIG. 4 illustrates the structure of a sequence header of ihe 
MPEG II video sequence of FIGS. 3a and 36. $5 

FIG. 5 illustrates the structure of a picture header of the 
MPEG II video sequence of FIGS. 3a and 36. 
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